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I. Real Party in Interest 

The real party in interest with respect to this application is Hewlett-Packard. 

II. Related Appeals and Interferences 

There are no other prior and pending appeals, interferences or judicial 
proceedings known to the Appellants, the Appellants 1 legal representative, or the 
assignee which may be related to, directly affect or be directly affected by or have a 
bearing on the Board's decision in this pending appeal. 

III. Status of Claims 

The claims currently pending in this application are Claims 1-20, all of which 
stand finally rejected. Claims 1-20 are being appealed. 

IV. Status of Amendments 

No amendments were filed after final rejection. 

V Summary of Claimed Subject Matter 

A method of visualizing and retrieving a data file from a set of data files, e.g., 
as illustrated in Fig. 1, is disclosed. Such a method is encompassed by claim 1 and 
includes the Fig. 1 steps of displaying, in block 110, a plurality of images 
representing corresponding data files on a display device using a first distance 
metric between each data file (e.g., page 5, paragraph [0011], lines 20-22), 
redisplaying, in block 120, a portion of the images on the display device using a 
refined distance metric (e.g., page 6, paragraph [0011], lines 1-3), and performing, in 
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block 140, at least one of retrieving, marking, and selecting at least one desired data 
file (e.g., page 6, paragraph [0011], lines 3-13). 

As discussed on specification page 16, lines 6-8, as the level of redisplay 
increases (e.g., the number of iterative redisplay operations is increased), the 
percentage of data used to calculate the distance metric increases (that is, "refining 
the distance metric"). Appellants' specification describes examples of how a more 
refined calculation is performed to redisplay a portion of the images that were initially 
displayed using a first distance metric between each data file (see, for example, the 
color histogram example of paragraph [0032]). Figs. 2A-2E illustrate a graphical 
representation of a display of images (Fig. 2a) using a first distance metric between 
each data file and an interactive selection by the user of an area 202 within the 
displayed images. Fig. 2B shows a redisplayed portion of the images within area 
202, produced using a refined distance metric. 

Claim 17 is directed to a method of interactively retrieving a data file from a 
set of data files in real time, e.g., as illustrated in Fig. 3. The claim 17 method 
encompasses the Fig. 3 steps of displaying a plurality of images in block 310, each 
image corresponding to a data file, on a display device using a first distance metric 
between each data file, interactively selecting in block 320, by a user, a portion of the 
images, redisplaying in block 330 the portion of the images in real time on the 
display device using a refined distance metric, and retrieving a desired data file in 
block 350 (e.g., paragraphs [0018] and [0019]). 

Thus, like claim 1, claim 17 recites performing a redisplay operation on 
images, originally displayed based on a distance metric between each data file using 
a "redefined distance metric." In addition, claim, 17 recites that this occurs in real 
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time, and that it is performed interactively by having the user actually select a portion 
of the images. 

VI. The Issues 

The final Office Action presents the following issues for a review on this 
appeal: 

A. Whether claims 1-7, 11, 12 and 16-19 are anticipated by U.S. 
Patent No. 6,240,423 (Hirata); 

B. Whether claims 8, 13 and 14 are unpatentable over the Hirata 
patent, and further in view of U.S. Patent 6,584,221 (Moghaddam et al.); 

C. Whether claims 9, 10 and 20 are unpatentable over the Hirata 
patent, and further in view of U.S. Patent 6,121,969 (Jain et al.); and 

D. Whether claim 15 is unpatentable over the Hirata patent, and 
further in view of U.S. Patent 5,528,259 (Bates et al.). 

VII. Argument 

A. The Hirata Patent Does Not Disclose Each And Every Element 
Recited In The Claims. 

The rejection based on the Hirata patent cannot stand because the Hirata 
patent fails to either expressly or inherently disclose a number of features recited in 
independent claims 1 and 17. Additionally, the dependent claims recite 
combinations of features defining combinations of separately patentable subject 
matter not disclosed in the Hirata patent. To anticipate the claimed invention, the 
cited document must disclose each and every feature set forth in the claimed 
combination of features (see, MPEP § 2131). 
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Independent claims 1 and 17 recite, among other features, redisplaying a 
portion of the images on the display device using a refined distance metric. In 
contrast, the Hirata patent is directed to a query based image matching system, 
wherein images are displayed only a single time per query. Using this method, the 
Hirata patent displays the results of a database query based upon results of a 
combined region based image matching and a boundary based image matching. 

With respect to claim 1 , the Examiner asserts on pages 2 and 3 (paragraph 3) 

of the final Office Action that Hirata patent discloses: 

Displaying a plurality of images representing corresponding data files 
on a display device using a first distance metric (Calculating a first 
similarity between a query image and images in the database using a 
region-based matching to produce a first set of similar images. Images 
being retrieved in order based upon the distance between the query 
image and the result candidate images, column 6) between each data 
file (e.g., Based on the mutual similarities between the images, retrieval 
results of the candidate images are sorted. The images which are 
similar to each other are assumed to be under one group and are re- 
ordered based on the similarity and users can specify the forms of the 
output for display; figures 1-11; column 6-9); 

Redisplaying a portion of the images on the display device using a 
refined distance metric (e.g., Hirata teaches using a distance metric 
such as a second similarity to retrieve images based on the set of 
similar images retrieved using a first similarity. The second similarity is 
now a "refined" distance metric because it refines the search for 
images. The second similarity of Hirata thus meets the claim limitation 
of a "refined distance metric." Here, a refined set of similar images is 
produced from the first set of similar images produced using a first 
distance metric, figures. 9, 11 and 12; column 6-9); and 

Performing at least one of retrieving, marking and selecting at least one 
desired data file (Hirata teaches retrieving and selecting at least one 
desired data file; see, figures 1-11; column 6-9). (Underlining and 
italics in original.) 
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With respect to claim 17, on page 6 of the Office Action, in numbered 

paragraph 4, the Examiner asserts that the Hirata patent discloses clustering image 

data files using a distance metric such as a first similarity, with reference to Figures 

1-11 and at column 6-9. The Examiner also asserts that the Hirata patent discloses: 

Interactively selecting, by a user, a portion of the images (e.g., a user 
[specifies] the regions of the images; e.g., the region division of figure 
3; figures 1-11; column 6-9); 

Redisplaying the portion of the images in real time on the display 
device using a redefined distance metric (e.g., Hirata teaches using a 
distance metric such as a second similarity to retrieve images based on 
the set of similar images retrieved using a first similarity. The second 
similarity is now a "refined" distance metric because it refines the 
search for images. The second similarity of Hirata thus meets the 
claim limitation of "a refined distance metric." Here, a refined set of 
similar images is produced from the first set of similar images produced 
using a first distance metric. Re-order based on the similarity among 
the candidate images of figure 9 or 2 nd stage image matching based on 
boundary of figure 1 1 and grouping and re-ordering is based on the 
similarity among candidates using the refined distance metric of figure 
1 1 ; figures 1-11; column 4); and 

Retrieving a desired data file (retrieved data file is displayed on a 
display device; figures 1-11; column 4). 



The foregoing rejections are respectfully traversed, as the Hirata patent, 
considered alone, or in combination with the various secondary references relied 
upon by the Examiner, fails to teach or suggest all features set forth in Appellants' 
independent claims 1 and 17. 

The present application discloses a multi-media database and classification 
system, which can, for example, provide for automatic classification and retrieval of 
multimedia files based on features of the multimedia files. Figure 1 shows an 
exemplary method for visualizing and retrieving data files in accordance with an 
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exemplary embodiment of the present invention. In step 1 10, a plurality of images 
representing data files on a display device are displayed using a first (i.e., coarse) 
distance metric. The distance metric represents a distance between each data file. 
In step 120, a portion of the images can be redisplayed on the display device using 
a refined distance metric. As described on specification page 6 beginning with line 
4, the distance metric can be refined at each of plural redisplays until a desired data 
file is found or a maximum refined distance metric is reached. In step 140, a desired 
data file can be retrieved, and/or can be marked or selected. 

Figures 2A-2E show a graphical representation of displays generated in 
accordance with exemplary embodiments, where each portion of the images 
redisplayed can be graphically selected by a user. Figure 2A is a screen capture of 
an exemplary display showing a first level of images representing data files on a two- 
dimensional display. A first, coarse, distance metric can be calculated, and allows 
the user to receive useful information about the organization of images on the 
display. Based on perceived properties of a desired image, a user can select an 
area 202 of the screen where a desired image most likely resides. Figure 2B shows 
a portion of images redisplayed as selected by area 202. Here the distance metric 
has been recalculated using more of the image information than was used in the first 
distance calculation. 

The redisplay process can be repeated, as represented by Figures 2C and 
2D. Distance metrics between images are again recalculated using more image 
feature data than previously used with respect to Figure 2B. Following the redisplay 
based on recalculated distance metrics, a desired data file (e.g., image 240) is 
identifiable and can be selected and retrieved as shown in Figure 2E. Figure 3 
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shows another exemplary method which can be implemented in accordance with the 
present invention, and Figure 4 shows a flowchart for performing a coarse to fine 
distance calculation. 

The foregoing features are broadly encompassed by independent claims 1 
and 17. In contrast to using query based image retrieval, exemplary embodiments of 
the present invention display images using a first distance metric between each 
data file, and provide for redisplaying a portion of the images using a refined 
distance metric. The claim 1 method includes a step of displaying a plurality of 
images representing corresponding data files on a display device using a first 
distance metric between each data file. Claim 1 also recites redisplaying a portion 
of the images on the display device using a refined distance metric. Claim 17 is 
directed to a method of interactively retrieving a data file from a set of data files in 
real time, and includes features similar to those mentioned with respect to claim 1. 

The foregoing features are neither taught nor suggested by the Hirata patent 
considered individually or in combination with the various secondary documents 
relied upon by the Examiner. 

In contrast to the presently claimed invention, the Hirata patent is directed to a 
method for querying a database of images and displaying the results in a manner as 
illustrated, for example, in Figure 10B of the Hirata patent. The results of the query 
are based upon a combination of region based image matching and boundary based 
image matching as described, for example, at column 2, lines 48-65. In the Hirata 
patent's query-based system, an initial query is submitted and a set of results is 
clustered. The clustering of the search results is performed using a distance 
between each file and a distance to the query, as discussed at column 6, lines 37- 
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41. In all cases, clustering involves, at least in part, using the distance to a query 
(see the Hirata patent's Figures 8, 9 and 10). The Hirata patent is not directed to 
redisplaying a portion of displayed images using a refined distance metric. In 
contrast to Appellants' claimed use of a refined distance metric to redisplay a portion 
of displayed images, Hirata teaches applying a new distance metric to all images 
during each image matching operation. 

Thus, in contrast to the method disclosed and recited in Appellants' 
independent claims 1 and 17, the Hirata patent fails to teach or suggest using a 
refined distance metric to redisplay a portion of displayed images. 

In addition, the Hirata patent fails to teach or suggest the claim 17 feature of 
interactive user selection for selecting a subset of images to be displayed. Rather, 
the query based system of the Hirata patent groups all images, and only creates new 
groupings based on a new query. The Hirata patent does not teach or suggest 
applying a refined distance metric in a second stage of image matching to redisplay 
a portion of displayed images. 

In rejecting claim 17 on page 6 of the Office Action, the Examiner refers to the 
region division in Figure 3 of the Hirata patent as illustrating user specified regions. 
However, Hirata does not teach or suggest that a refined distance metric, as 
presently claimed, is used to redisplay the images of any defined region (see Hirata 
at column 4, line 10 to column 5, line 13). As such, the Hirata patent fails to teach or 
suggest the user interactive selection feature of Appellants' claim 17. 

Thus, the Hirata patent fails to teach or suggest the features recited in 
Appellants' independent claims 1 and 17. 
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As such, independent claims 1 and 17 are allowable. All of the other claims 
depend from independent claims 1 and 17 and recite additional advantageous 
features which further distinguish over the documents relied upon by the Examiner. 
Hence, for these additional reasons, the Hirata patent fails to disclose each and 
every limitation set forth in independent claims 1 and 17, and hence also in claims 
depending from these independent claims. 

For at least these reasons, the rejection of independent claims 1 and 17 is 
improper. Accordingly, the rejection should be reversed. 

B. The Hirata Patent And The Secondary References, Fail To Teach Or 
Suggest All Features Of Claims 1 and 17 

The Hirata patent, considered alone, or in combination with the various 
secondary references relied upon by the Examiner, fails to teach or suggest all 
features set forth in Appellants' independent claims 1 and 17. Specifically, the Hirata 
patent fails to teach or suggest at least the above argued features of redisplaying a 
portion of the images on the display device using a refined distance metric recited in 
Appellants 1 independent claims 1 and 17. In contrast, the Hirata patent is directed to 
a query based image matching system, wherein images are displayed only a single 
time per query. 

The patents to Moghaddam et al., Jain et al. and Bates et al., as relied upon 
by the Examiner, fail to overcome the noted deficiencies of the Hirata patent. For 
example, the Moghaddarm et al. patent is directed to a method for representing an 
image in an image retrieval database, further disclosing separation and filtering of 
given images to extract color and texture features; the Jain et al. patent is directed to 
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a visual navigation in perceptual databases, the given images being ranked and 
correlated to allow perception of selected images and the relationship between the 
images; and the Bates et al. patent discloses multi-dimensional scrolling of 
overlapping data collections which are displayed in multiple layers. The patents to 
Moghaddam et al., Jain et al. and Bates et al., as relied upon by the Examiner, do 
not disclose or suggest at least the above argued features of redisplaying a portion 
of the images on the display device using a refined distance metric recited in 
Appellants' independent claims 1 and 17. 

As such, independent claims 1 and 17 are allowable. Claims 8-10 and 13-15 
depend from claim 1, and claim 20 depends from claim 17, and recite additional 
advantageous features which further distinguish over the documents relied upon by 
the Examiner. 

Claims 8, 13 and 14 stand rejected under 35 U.S.C. § 103(a) as being 
unpatentable over the Hirata patent, and further in view of U.S. Patent 6,584,221 
(Moghaddam et al.); claims 9, 10 and 20 stand rejected under 35 U.S.C. § 102(e) as 
being unpatentable over the Hirata patent, and further in view of U.S. Patent 
6,121,969 (Jain etal.); and claim 15 stands rejected under 35 U.S.C. § 103(a) as 
being unpatentable over the Hirata patent, and further in view of U.S. Patent 
5,528,259 (Bates et al.). 

For at least these reasons, the Hirata patent does not disclose or suggest 
each and every element of claims 8-10, 13-15 and 20, and the patents to 
Moghaddam et al., Jain et al. and Bates et al. do not make up for the deficiencies of 
the Hirata patent. Accordingly, the rejection should be reversed. 
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C. The Examiner's Assertions In The Advisory Action Are Traversed 
Because The Hi rata Patent Fails To Disclose Or Suggest Each And 
Every Element Recited In The Claims. 

The Examiner responds to these points at page 2 of the Advisory Action: 

1) Applicant argues in essence that the Hirata patent is not 
directed to redisplaying a portion of displayed images using a refined 
distance metric. In response, Hirata teaches in column 7 that the step 
of image matching based on boundary is applied to the results of 
image matching based on region and the step of grouping and re- 
ordering based on similarity among result candidates is applied to the 
results of the image matching based on boundary. Moreover, in 
column 14 or in the claims 37-39, Hirata clearly teaches displaying the 
second set of images grouped by the clusters on the display and 
REDISPLAYING RE-ORDERED SECOND SET of IMAGES ON THE 
DISPLAY and therefore at least two steps of displaying is involved in 
the process. 

2) Moreover, Hirata teaches in column 6 a [similarly] matching 
among the candidate images resulting from the region-based and 
boundary-based matching. Not only users can select regions for the 
image matching or querying, but also users can specify other 
parameters such as the threshold values to determine how the images 
are [categorized] into groups and this categorization is used to 
REDUCE the number of images to be displayed. 

3) Hirata discloses in column 7 that the number of results (the 
number of the input images for the step (3) as taught in Hirata) to be 
processed in the step of image matching based on boundary and the 
step of grouping and re-ordering based on similarity among result 
candidate images can be CONTROLLED BY THE USER, and such a 
selection further limits the number of the candidate images to be re- 
displayed in step (3). 
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4) The displaying and re-displaying of the resulting candidate 
images are [interactive] because users can select the number of the 
candidate images to be processed and the threshold values for the 
clustering of the candidate images, thereby determining how the 
portion of candidate images (which is less than the candidate images 
from step (1) and (2) as taught in column 7 of Hirata) are arranged on 
the display. By selecting the number of the candidate images in the 
step of grouping and re-ordering based on similarity among the result 
candidates and the selection of the representative image in each 
cluster, the resulting images being redisplayed is less than the set of 
the resulting candidates from steps (1) and (2). Hirata thus teaches 
redisplaying a portion of displayed images selected in the previous step 
of the boundary image matching using a mutual similarity measure. 
Users' selection of the candidate images from the steps (1) and (2) 
produces a refined set of the candidate images for the step (3). Finally, 
it is noted that the mutual similarity as calculated in Hirata's column 9, 
lines 1 — the claim limitation of a distance measure. 

5) From the teaching of column 6-7 and 14, it is concluded that 
Hirata at least discloses two separate steps of displaying the candidate 
images wherein a lesser number of the clustered candidate images can 
be re-ordered and re-displayed. Hirata further teaches interactive user 
selection for selecting a subset of images to be re-ordered and re- 
displayed based on the mutual similarity. 

The foregoing assertions by the Examiner are respectfully traversed, as the 
Hirata patent, considered alone, or in combination with the various secondary 
references relied upon by the Examiner, fails to teach or suggest all features set 
forth in Appellants' independent claims 1 and 17. The image matching based on 
boundary, as taught by the Hirata patent, does not redisplay a portion of the images 
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as claimed, and does not suggest redisplay of a portion of the images first displayed. 
Rather, the first and second similarity calculations disclosed by the Hirata patent are 
a mere sequence of calculations in a method for querying a database. 

Moreover, a similarity matching among candidate images resulting from the 
Hirata patent's region-based and boundary-based matching does not produce a 
redisplay of a portion of the already displayed images for retrieving. The Hirata 
patent does not disclose a first visual display of a plurality of images, and 
redisplaying a portion of the first displayed images, the portion being in reference to 
the first display. 

The Examiner asserts that the displaying and re-displaying of resulting 
candidate images are interactive because users can select the number of the 
candidate images. However, the selection criteria are not based upon visual 
inspection of the first display and defining a portion of the first display for redisplay. 
The Hirata patent's clustering does not cure the noted deficiencies, because the 
clustering is merely a process in the second similarity calculation, but does not teach 
or suggest at least the claimed feature of redisplaying a portion of the first displayed 
images. The Hirata patent does not teach or suggest at least these salient claim 
features. 

For at least the reasons discussed above, it thus respectfully submitted that 
the claims are in condition for allowance. 

VIII. Conclusion 

For the reasons discussed above, appellants respectfully submit that the 
Examiner's decision finally rejecting Claims 1-20 should be reversed. 
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IV Evidence Appendix 

See attached Evidence Appendix for copies of evidence relied upon by 
Appellant. 

NONE 



X Related Proceedings Appendix 

See attached Related Proceedings Appendix for copies of decisions identified 
in Section II, supra . 



NONE 



Date June 22. 2005 



Respectfully submitted, 

Burns, Doane, Swecker& Mathis, L.L.P. 




By: 



Patrick C. Keane 
Registration No. 32,858 



P.O. Box 1404 

Alexandria, Virginia 22313-1404 
(703) 836-6620 
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VIII. CLAIMS APPENDIX 

The Appealed Claims 

Claim 1 . A method of visualizing and retrieving a data file from a set of 
data files comprising: 

displaying a plurality of images representing corresponding data files on a 
display device using a first distance metric between each data file; 

redisplaying a portion of the images on the display device using a refined 
distance metric; and 

performing at least one of retrieving, marking, and selecting at least one 
desired data file. 

Claim 2. The method of claim 1 , further comprising: 

repeating the redisplaying step until a desired data file is identifiable. 

Claim 3. The method of claim 1 , wherein the first distance metric is 
calculated by a method comprising: 

computing a feature vector for each data file; and 

calculating the first distance metric between each data file using a first subset 
of data contained in the feature vector. 
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Claim 4. The method of claim 3, wherein the refined distance metric is 
calculated by a method comprising: 

calculating a second distance metric between each data file using a second 
subset of data contained in the feature vector which is greater than the first subset. 

Claim 5. The method of claim 4, wherein the step of computing the 
feature vector comprises: 

computing the feature vector for each data file before starting the method; 

storing the feature vector for each data file; and 

accessing the feature vector for each data file. 

Claim 6. The method of claim 4, wherein each feature vector has a length 
of at least eight. 

Claim 7. The method of claim 4, wherein the feature vector includes at 
least one of a color feature and a texture feature. 

Claim 8. The method of claim 4, wherein the feature vector includes at 
least one of a color histogram, color moment, color coherence histogram, 
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Multiresolution Simultaneous Autoregressive (MRSAR) Model, coarseness, and 
directionality. 

Claim 9. The method of claim 1 , wherein the first distance metrics are 
mapped into an N-dimensional space using FastMap for displaying and wherein the 
refined distance metrics are mapped into an N-dimensional space using FastMap for 
redisplaying. 

Claim 10. The method of claim 9, wherein N is two or three. 

Claim 1 1 . The method of claim 1 , wherein the data files are image files. 

Claim 12. The method of claim 1 , wherein the data files are video files. 

Claim 1 3. The method of claim 1 , further comprising: 

establishing a fixed scale that spans a maximum distance between the 
plurality of data files; and 

indicating a relative position on the fixed scale for the redisplay of the portion 
of the images, thereby providing the user with a reference frame. 
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Claim 14. The method of claim 1 , wherein the fixed scale is at least one of 
a linear scale, a logarithmic scale, and a hyperbolic scale. 

Claim 1 5. The method of claim 1 , further comprising: 

providing a display depth indication that represents an amount of overlapping 
of images on the display; and 

scrolling to view images that were previously not viewable due to overlapping 
of the images. 

Claim 16. The method of claim 1, wherein the portion of the images 
redisplayed is graphically selected by the user. 

Claim 17. A method of interactively retrieving a data file from a set of data 
files in real time comprising: 

displaying a plurality of images, each image corresponding to a data file, on a 
display device using a first distance metric between each data file; 

interactively selecting, by a user, a portion of the images; 

redisplaying the portion of the images in real time on the display device using 
a refined distance metric; and 

retrieving a desired data file. 
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Claim 18. The method of claim 17, wherein the first distance metric is 
calculated by a method comprising: 

computing a feature vector for each data file; and 

calculating the first distance metric between each data file using a first subset 
of data contained in the feature vector. 

Claim 19. The method of claim 18, wherein the refined distance metric is 
calculated by a method comprising: 

calculating a second distance metric between each data file using a second 
subset of data contained in the feature vector which is greater than the first subset. 

Claim 20. The method of claim 19, wherein the first distance metrics are 
mapped into a N-dimensional space using FastMap for displaying and second 
distance metrics are mapped into a N-dimensional space using FastMap for 
redisplaying. 
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IX. EVIDENCE APPENDIX 

NONE 
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